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(57) Abstract: A method of synchronizing a client with a media signal includes receiving one or more actions corresponding to the 
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SYNCHRONIZING A CLIENT WITH A MEDIA SIGNAL 

5 Background 

The Internet and traditional radio and TV broadcasts both 
offer users/viewers access to a wide variety of information. 
Both mediums, generally, have different strengths. For example, 
the Internet enables users to view the content offered by a site 

10 at their own pace. Additionally, many sites provide a degree of 
inter-activity absent from traditional TV and radio broadcasts. 
For example, a web-page may solicit user input via forms or 
"applets" and respond accordingly. While Internet sites have 
greatly improved the quality and speed of network video 

15 transmission, traditional broadcasts generally retain the upper 
hand in image and sound quality. 



Suinmary 

In general, in one aspect, the invention features a method 
20 of synchronizing a client with a media signal. The method 
includes receiving one or more actions corresponding to the 
media signal, determining an application for handling the 
actions, and causing the determined application to handle at the 
actions . 
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Embodiments may include one or more of the following 
features. The media signal may be an audio signal and/or a 
video signal. Determining an application can include parsing 
action information. For example, parsing action information can 
5 include determining if the action information corresponds to a 
ITRI (Universal Resource Indicator) or a command. For actions 
including a URI, the method can include navigating a browser to 
the URI. 

The method may further include prompting a user before 
10 causing the determined application to handle the action. The 
method may also include queuing the action. The action may 
include an indication of when the determined application should 
handle the action (e.g., an SMPTE (Society of Motion Pictures 
and Television Engineers) time-code) . The actions may be 
15 received from a server. 

The method may also include receiving criteria associated 
with the action and evaluating the criteria. The criteria may 
include one or more user characteristics (e.g., age, location, 
and gender) . 

20 In general, in another aspect, the invention features a 

computer program product, disposed on a computer readable 
medium, the computer program for synchronizing a client with a 
media signal. The program includes instructions for causing a 
processor to receive one or more actions corresponding to the 
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media signal, determine an application for handling at least one 
of the one or more received actions, and cause the determined 
application to handle at least one of the one or more received 
actions . 

5 In general, in another aspect, the invention features a 

method of synchronizing one or more clients with a media signal. 
The method includes receiving information identifying an action 
corresponding to the media signal, receiving information 
identifying criteria for the action, and transmitting the action 

10 and criteria toward the one or more clients. 

Embodiments may include one or more of the following. The 
action includes an identification of a time for processing the 
action and, for example, a URI or some application command or 
data. Transmitting the action and criteria toward one or more 

15 clients may include transmitting the action and criteria to a 
server and determining at the server, based on the action 
criteria, whether to transmit the action to a client. Such a 
determination may be done based on user characteristics 
associated with the client. In some embodiments, a client 

20 receives the. action and determines, based on the action 
criteria, whether to execute the action. 

In general, in another aspect, the invention features a 
method of synchronizing a client with a media signal. The 
method includes receiving one or more actions corresponding to 
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the media signal, the actions including an indication of a time 
to execute each action. The method also includes determining a 
delay time for the action and handling the action in accordance 
with the delay time. 
5 Embodiments may include one or more of the following 

features. Determining a delay time may be based on the location 
of a client and/or on a broadcast medium used to transmit the 
media signal to a household of the client. 

Advantages will become apparent in view of the following 
10 description, including the figures and the claims. 

Brief Description of the Drawings 
PIGS. lA and IB are diagrams of systems for synchronizing a 
client with a media signal. 
15 FIG. 2 is a flow- chart of a process for transmitting 

actions to a server. 

FIG. 3 is a screenshot of a user interface for identifying 
actions corresponding to a media signal. 

FIG. 4 is a flow- chart of a process for transmitting 
20 actions to clients. 

FIG. 5 is a diagram of a client including instructions for 
processing actions . 

FIG. 6 is a flow- chart of a process for receiving and 
processing actions . 
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Detailed Description 
FIG. lA shows an example of a system 10 0 for synchronizing 
actions performed by computer 106 with a TV 104 or radio show. 
5 For example, as a station 108 broadcasts a TV show (e.g., via 
airways, cable, or satellite), a computer 106 web-browser may 
automatically navigate to Internet web- sites corresponding to 
the TV show content. That is, when the TV show mentions the 
U.S. Soccer Team, a user' s ^computer may automatically navigate a 

10 web-browser to a URI (Universal Resource Indicator) of 

"www.soccer.com/usa/schedule". Similarly, when the TV show 
mentions a particular player, a user's web-browser may 
automatically navigate to a URI with a detailed biography. The 
system 100 can support a wide variety of applications. For 

15 example, inter-active web-pages may quiz and provide feedback to 
students viewing a broadcast school lesson. 

Though the system 100 can provide Internet access that ia 
synchronized with a media signal, the system 100 need not be 
limited to Internet navigation. For example, the system 100 can 

20 feature actions that transmit data and/or invoke different 
client 106 applications other than a browser. For example, 
actions may cause a client 106 to download and install 
protection software when a TV program reports the latest 
computer virus. 
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In addition to synchronization of clients 106 with a 
signal, the system 100 can also tailor the actions performed by 
the clients 10 6, for example, based on user characteristics 
and/or preferences. That is, an action may cause a computer to 
5 present an adult with an option to charge an item shown on TV to 
their credit card, but not present this option to a child. To 
provide this feature, the server 112 and/or client 10 6 may store 
user characteristics such as an address, gender, age, and so 
forth. 

10 The system 100 can be implemented in a variety of ways. 

For example, the system 100 of FIG. lA includes a control 
computer 110 that enables an operator to define actions 
corresponding to a media signal 114 (e.g., TV or radio signal) . 
The system 100 also includes a server 112 that receives actions 

15 from the control computer 110 and transmits (e.g., multicasts) 
the actions to clients 106 in households 102 viewing or 
listening to a particular broadcast. 

In an alternate implementation shown in FIG. IB, the 
control computer 110 can embed actions in the signal itself, for 

20 example, in .a VBI (Vertical Blanking Interval) of a video 

signal. Thus, instead of network transmission of the actions 
from the control computer 110 to the server 112, the server 112 
can monitor the signal as broadcast, extract the actions, and, 
again, transmit them to the corresponding client 106. 
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Additionally, clients 106 can monitor a broadcast or other media 
signal to monitor for actions without the seorver 112 . 

Though shown as personal computers 106 in FIG. lA and IB, 
client software and/or hardware can reside in a wide variety of 
5 network devices such as network appliances, handheld PCs, PDAs 
(Personal Digital Assistants), game consoles, Internet phones, 
WAP (Wireless Access Protocol) devices, set-top boxes, 
ReplayTV®, TiVo®, and so forth. Additionally, though FIG. 1 
shows a TV 104 as an example of an a device that presents a 

10 media signal, the media signal may also be presented by the same 
device arunning the client software. For example, the media 
signal may be streaming audio or video carried across the 
Internet to a client computer. Further, while shown as being 
received from a remote location, the media signal may come from 

15 a local source. For example, a VCR may output a media signal to 
for display and correlation with different actions downloaded 
from a seirver or from within the VCR signal. 

As shown in FIG. 2, an operator can use the control 
computer 110 to define 205 actions and specify how the actions 

20 should synchronize with the media signal (e.g., when the actions 
should occur relative to the start of the signal) . Each action 
defines some operation the client should perform. For example, 
an action may include a command line identifying an application 
(e.g., "'run virus_install . exe" ) / a URI (e.g., www.tvshow.com) 
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specifying a web- site a browser should visit, data representing 
the real or virtual coordinates of an object or person of a TV 
show for use by a game application, and so forth. The control 
computer 110 can, in turn, transmit 204 the actions over a 
5 secure (e.g., encrypted) connection (e.g., a TCP/IP 

(Transmission Control Protocol/Internet Protocol) connection) 
established 202 with the server 112. This encryption may be 
based upon a security key. The key can be changed at the 
administrative level. Once the key is decoded by the server, an 

10 action can be queued- The system may also employee an 

encryption key at the end user level. As an additional measure 
of security, the system can also make use of firewall security 
at the server/control level as well as the client level. 

The control computer 110 can offer a variety of different 

15 techniques for defining actions. For example, the control 

computer 110 can enable an operator to define actions in real- 
time. That is, as a signal broadcasts, an operator can define 
actions for the broadcast and immediately send them to the 
server 112 for transmission to clients 106. This can enable an 

20 operator to synchronize clients 106 during live broadcasts. The 
same technique may also enable an operator to specify 
transmission delays. For example, an operator may define an 
action and specify that it not be transmitted or executed by the 
clients 106 for a period of i2-seconds. 
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The user interface techniques described above feature an 
operator sitting and viewing the broadcast in real-time and 
preparing appropriate client computer actions. While the 
control computer 110 may ease the demands on an operator, for 
5 example, by providing previously defined actions for selection, 
the process of defining actions in real-time may require some 
dexterity and quick-wittedness on the part of the operator to 
keep up with the on-going broadcast. Thus, the control computer 
110 may provide a user interface that enables the operator to 

10 define actions for pre-recorded material at his or her leisure. 

Thus, the control computer 110 may provide a user interface 
that enables an operator to define an action and associate the 
action with a particular time-code. For example, video 
typically includes embedded SMPTE (Society of Motion Picture and 

15 Television Engineers) time-codes identifying each video frame. 
Thus, the user interface can enable an operator to associate an 
action with a particular time-code. For example, an action of 
^^Get Smart: 000100 : www.getsmart.com/episodel2" may define an 
action that causes a client computer's 106 web-browser to visit 

20 the ^^www. get smart .com" web- site at the one-hundredth frame of 
Episode 12 of ^"Get Smart" . The control computer 110 may use 
other time-codes other than SMPTE (e.g., NNTP time - codes ) . 
Additionally, the operator may simply enter a time offset for 
the action (e.g., one minute after broadcast begins). 
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Again, instead of, or in addition to, transmitting actions 
to a server 112 via a network connection, the control computer 
110 may embed the actions within a media signal such as the 
video blanking interval (VBI) of a video. In such embodiments, 
5 a server 112 can monitor the media signal, extract action 
definitions, and transmit the actions to clients 106. 
Additionally, in implementations that embed the actions in the 
video and/or audio signal, a system 100 need not use a server 
112 at all. For example, client computers 106 can monitor media 

10 signals received by the household 102 TVs 104 and process the 
embedded actions. 

FIG. 3 illustrates an example of a user interface 300 
presented to an operator of a control computer 110. As shown, 
the interface 3 00 enables an operator to define actions 3 02 for 

15 a particular broadcast. The action can include a URI, an 
application program command and parameters, and/or other 
instructions or data for transmission to the clients 106. The 
action can also include synchronization information (e.g., a 
date, time, and/or frame 110 of an action) . The interface 300 

20 may also enable an operator to define whether the action is 

performed automatically or whether a user at the client computer 
106 is prompted before execution. The interface 3 00 may further 
enable an operator to define criteria that must be satisfied for 
performance of the action. For example, the operator may 

10 
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construct a Boolean expression of user characteristics (e.g., 
(gender = male) AND (age > 35) ) . The operator may define 
multiple actions for the same signal time -code with different 
actions having different criteria- This enables the operator to 
5 tailor the behavior of client computers 110 for different 

classes of users. As shown, the interface 300 also includes a 
list 3 04 of previously defined actions for selection and a list 
306 of actions already assigned to the broadcast. 

FIG. 4 shows a process 400 that a server 112 can use to 

10 transmit 408 actions received 402 from a control computer 110 or 
detected in a signal (e.g'. , the VBI portion of a video). After 
receiving the actions, the server 112 can log the actions. For 
example, the server 110 may dynamically add a description of 
broadcast actions to an HTML (HyperText Markup Language) or XML 

15 (extensible Markup Language) page listing actions for a 

particular show or broadcast. Users and contributors can access 
the logs to view actions for a particular broadcast long after 
the show has aired. 

Before and during a broadcast, the server 110 may receive 

20 registration requests from different system 100 users. 

Registration identifies a broadcast the user is viewing and, 
potentially, the IP (Internet Protocol) address of their client. 
Additionally, for new members, registration may involve 
providing demographic data such as an address, age, gender. 
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income, and so forth. This data may be stored by the server 110 
to tailor the actions delivered to a user's client 106. For 
existing members, registration may involve entering a user name 
so the server 110 can retrieve the previously stored demographic 

5 data for the user. 

Registration and other interaction with the se3nrer 112 may 
occur via the Internet (e.g., a login web-page, a preferences 
web-page and so forth) or via a telephone voice response system 
that enables a user to customize and select different services 

10 transmitted to the client. 

During broadcast of a TV or radio program, the server 112 
sends actions for the broadcast to clients 10 6 identified as 
viewing the show. For example, the server 112 can multicast 
the actions using UDP (User Datagram Protocol) or other 

15 protocols. Again, the server 112 may select actions for 

transmission based on user demographic data. The server 112 may 
handle timing issues by transmitting actions at the specified 
times. Alternatively, the server 112 may include 
synchronization information in the action and leave it to the 

20 client 106 to process the action at the designated time. 

The same show may be broadcast at slightly different times. 
For example, a satellite transmission generally lags cable 
transmission by tens of seconds. Additionally, different 
geographic locations broadcast at different times. The server 

12 
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112 can offset 408 transmission of actions based on the 
geographic location of a user (e.g., based on the user's zip 
code) and/or transmission sources (e.g., cable or satellite). 

Communication between the clients 106 and server 110 need 
5 not be one way. That is, the clients 106 may send data to the 
server 110, for example, using TCP/IP. This enables the server 
110 to monitor user behavior and can enable the server to select 
actions based on prior user behavior. 

FIG. 5 shows an example of a client 106. As shown, the 
IQ client 106 includes an operating system 504 (e.g., Microsoft® 
Windows 2000®) and one or more applications 502 (e.g., 
Microsoft® Explorer® and Microsoft® Word®) . The client 106 also 
includes instructions 500 for processing actions. As shown, the 
processing instaructions 500 can invoke other applications and 
15 can even make operating system 504 calls. 

FIG. 6 illustrates a process 600 the instructions (500 of 
FIG. 5) may implement. As shown, the process 600 may include 
registering the user with a server 602, for example, by 
navigating a browser to a log- in web- site associated with the 
20 server. Registering enables the client to identify a program 
being viewed or heard. Of course, a user need not actually be 
viewing or listening to a program. 

Thereafter, the client may receive a stream of one or more 
actions 604 corresponding to the identified program as the 

13 
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program broadcast progresses. The client determines an 
application for the action. For example, the client may parse 
information included in the action to identify whether an action 
includes a URI. If such parsing succeeds, the client can pass 

5 the URI to a browser. Alternatively, the client can look for 

names of applications and/or keyword such as "run" that indicate 
what operation the client should perform in response to the 
action (e.g., which application should handle the action). 
The system (s) described above can use a variety of 

10 architectures. For example, many features provided by the 
server can instead be provided by the client and vice-versa. 
For example, instead of selection by the server 110, the server 
110 may transmit action criteria for evaluation 60 6 by the 
client. Additionally, either the server 110 or client 106 may 

15 specify that the action be queued 610 or executed 612 upon 
receipt or arrival of a specified time. Further, either the 
server 110 or client 10 6 may request that the client prompt a 
user before executing an action. This can enable, for example, 
a user to finish reading information at one web- site before 

20 moving on to another. 



Implementations 

The techniques described herein are not limited to any 
particular hardware or software configuration; they may find 

14 
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applicability in any computing or processing environment . The 
techniques may be implemented in hardware or software, or a 
combination of the two. Preferably, the techniques are 
implemented in computer programs executing on programmable 

5 computers that each include a processor, a storage medium 

readable by the processor (including volatile and non- volatile 
memory and/or storage elements) , at least one input device, and 
one or more output devices . 

Each program is preferably implemented in high level 

10 procedural or object oriented programming language to 

communicate with a computer system. However, the programs can 
be implemented in assembly or machine language, if desired. In 
any case the language may be compiled or interpreted language. 
Each such computer program is preferably stored on a 

15 storage medium or device (e.g., CD-ROM, hard disk, or magnetic 
disk) that is readable by a general or special puaqpose 
programmable computer for configuring and operating the computer 
when the storage medium or device is read by the computer to 
perform the procedures described herein. The system may also be 

20 considered to be implemented as a computer- readable storage 

medium, configured with a computer program, where the storage 
medium so configured causes a computer to operate in a specific 
and predefined manner. 

15 
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Other embodiments are within the scope of the following 
claims . 



16 
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What is claimed is: 

1. A method of synchronizing a client with a media signal, 
the method comprising: 

receiving one or more actions corresponding to the media 

signals- 
determining an application for handling at least one of the 

one or more received actions; and 

causing the determined application to handle at least one 

of the one or more received actions. 

2. The method of claim 1, wherein the media signal 
comprises at least one of the following: an audio signal and a 
video signal, 

3. The method of claim 1, wherein determining an 
application comprises parsing action information. 

4. The method of claim 3, wherein parsing action 
information comprises determining if the action information 
corresponds to a URI (Universal Resource Indicator) . 



17 



wo 02/21840 



PCT/USOl/28246 



5. The method of claim 3, wherein parsing action 
information comprises determining if the action information 
corresponds to a command. 

5 6. The method of claim 1, wherein causing the determined 

application to handle at least one of the actions comprises 
causing an Internet browser to navigate to a URI specified in 
the action. 

10 7. The method of claim 1, further comprising prompting a 

user before causing the determined application to handle the 
action. 

8. The method of claim 1, further comprising queuing the 
15 action. 

9. The method of claim 1, wherein the action comprises at 
least one of the following: a URI and an application command. 

20 10. The method of claim 9, wherein the action comprises an 

indication of when the determined application should handle the 
action. 

18 
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11. The method of claim 10, wherein the indication 
comprises a SMPTE (Society of Motion Pictures and Television 
Engineers) time -code . 

12. The method of claim 1, wherein receiving the at least 
one actions comprises receiving the at least one actions from a 
server. 

13. The method of claim 1, further comprising 
receiving criteria associated with the action; and 
evaluating the criteria. 

14. The method of claim 13, wherein the criteria comprises 
one or more user characteristics. 

15. The method of claim 14, wherein the user 
characteristics comprises at least one of the following: age, 
location, and gender. 

16. A computer program product, disposed on a computer 
readable medium, the computer program for synchronizing a client 
with a media signal, the program comprising instructions for 
causing a processor to: 



19 
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receive one or more actions corresponding to the media 
signal ; 

determine an application for handling at least one of the 
one or more received actions; and 

cause the determined application to handle at least one of 
the one or more received actions. 

18 . A method of synchronizing one or more clients with a 
media signal, the method comprising: 

receiving information identifying an action corresponding 

to the media signals- 
receiving information identifying criteria for the action; 
transmitting the action and criteria toward the one or more 

clients . 

19. The method of claim 18, wherein the action comprises 
identification of a time for processing the action and at least 
one of the following: a URI and an application command. 

20. The method of claim 18, wherein transmitting the 
action and criteria toward one or more clients comprises 
transmitting the action and criteria to a server. 

21. The method of claim 20, further comprising 



20 
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determining at the server, based on the action criteria, 
whether to transmit the action to a client. 

22. The method of claim 21, wherein determining comprises 
determining based on user characteristics associated with the 
client . 

23. The method of claim 22, wherein the user 
characteristics comprise one or more of the following: age, 
gender, and location. 

24. The method of claim 18, further comprising 
receiving the action and criteria at a client; and 
determining at the client, based on the action criteria, 

whether to execute the action - 

25. The method of claim 24, wherein determining comprises 
determining based on user characteristics. 

26. The method of claim 25, wherein the user 
characteristics comprise one or more of the following: age, 
gender, and location. 
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27. A method of synchronizing a client with a media 
signal, the method comprising: 

receiving one or more actions corresponding to the media 
signal, the actions including an indication of a time to execute 
5 each action; 

determining a delay time for the action; and 

handling the action in accordance with the delay time. 

28. The method of claim 27, wherein determining a delay 

10 time comprises determining a delay time based on the location of 
a client. 

29. The method of claim 27, wherein determining a delay 
time comprises determining a delay time based on a broadcast 

15 medium used to transmit the media signal to a household of the 
client . 

30. The method of claim 27, wherein handling the action 
comprises determining when to transmit the action from a server 

20 to the client . 

31. The method of claim 27, wherein handling the action 
comprises determining when to execute the action at the client. 
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